Simulated large aperture lens

ABSTRACT

A camera apparatus includes a photogrammetric range sensor, small aperture lens and a processor configured to carry out a method for simulating a large aperture lens. The method includes capturing a photographic image data of a scene using the camera with the lens set to a small aperture size. Simultaneously or near-simultaneously with capturing the image, the camera captures photogrammetric range data pertaining to depth of objects in a field of view of the camera, using the range sensor. The processor then processes the photographic image data using the photogrammetric range data to obtain second photographic data simulating an image captured using a lens set to an aperture size larger than the small aperture size.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.14/557,334, filed Dec. 1, 2014, now U.S. Pat. No. 9,325,891, which is acontinuation of U.S. application Ser. No. 14/089,659, filed Dec. 2,2014, now U.S. Pat. No. 8,902,354, which is a continuation of U.S.application Ser. No. 13/430,630, filed Nov. 26, 2013, now U.S. Pat. No.8,593,565, which claims priority pursuant to 35 U.S.C. §119(e) to U.S.provisional application Ser. No. 61/467,829 filed Mar. 25, 2011, whichapplications are incorporated by reference herein.

FIELD

The present application relates to optics, and more particularly tocamera components and methods for simulating a large-aperture lens.

BACKGROUND

A well-defined focal plane resulting in blurring of background and/orforeground objects in photographs is one of the benefits of highquality, large aperture lenses. However, large apertures come withsignificant downsides. Such lenses are more expensive and difficult tomanufacture; are heavier and larger than comparable smaller aperturelenses, and can dramatically magnify the impact of even minor errors infocus. For example, using an expensive 85 mm F/1.2 lens, open to itsfull aperture, a photographer may find the depth of field so narrow thatif the focus is on the tip of the subject's nose, the eyes may be out offocus. Similarly, wildlife and sports photography are areas where anarrow depth of field delivers striking images of the player or animalisolated from its surroundings, but even a small amount of motion by thesubject after focus has been set can result in the desired area movingout of the area of sharp focus.

Modern digital cameras are capable of delivering low-noise images atrelatively high ISO, or light sensitivity, settings. Compared to film,or earlier digital sensors, modern sensors are many times morelight-sensitive—and light sensitivity in sensors is increasingsignificantly each year. Combined with modern post-processing softwareand techniques, one of the biggest benefits of large aperture lenses(note that “large aperture” literally refers to the size of theaperture; apertures are actually numbered in reverse, so that thelargest apertures have the lowest numbers, for example 1.2 is muchlarger than 8.0) is the ability to deliver a lot of light to the imagingsurface, allowing adequate light to be delivered even in low lightconditions. With each advance in sensor light sensitivity and noisereduction, the utility and value of this quality of large aperturelenses is reduced. Indeed, an F/4.0 lens mounted on a modern digitalDSLR can now deliver quality photographs in light conditions that mayhave required an F/2.8 lens just a few years ago.

As lens aperture is reduced (i.e. as the F/number increases), the depthof field increases. A “pinhole” aperture delivers a functionallyinfinite depth of field. While the photons can literally interfere witheach when the aperture becomes too small, cameras/lens combinationstypically deliver high quality photographs with enormous depth of fieldat small lens apertures, such as f/11 or f/22. Such photographs,however, can appear unnatural and flat, as objects both near and far arerendered with comparably sharp focus. It would be desirable, therefore,to overcome these and other limitations of the prior art with a newcamera system and method.

SUMMARY

Methods, apparatus and systems for simulating a large aperture lens aredescribed in detail in the detailed description, and certain aspects aresummarized below. This summary and the following detailed descriptionshould be interpreted as complementary parts of an integrateddisclosure, which parts may include redundant subject matter and/orsupplemental subject matter. An omission in either section does notindicate priority or relative importance of any element described in theintegrated application. Differences between the sections may includesupplemental disclosures of alternative embodiments, additional details,or alternative descriptions of identical embodiments using differentterminology, as should be apparent from the respective disclosures.

A camera configured to include an image processor and a photogrammetrydevice for sensing depth of field may perform a method for simulating alarge aperture lens. The camera may comprise a dedicated device, or acamera function in a more general-purpose electronic device, for examplea smart phone or notepad computer. The method may include capturing aphotographic image data of a scene using the camera having a lens set toa first aperture size. The method may further include simultaneously ornear-simultaneously with capturing the image, capturing photogrammetricrange data pertaining to depth of objects in a field of view of thecamera. The method may further include processing the photographic imagedata using the photogrammetric range data to obtain second photographicdata simulating an image captured using a lens set to a second aperturesize, wherein the second aperture size is wider than the first aperturesize. The method may further include storing the photographic image dataassociated with the measurement data in an electronic memory.

In an aspect, capturing the photogrammetric range data may furtherinclude illuminating the scene using a narrow-band light adjacent to thelens, and measuring intensity of narrowband light reflected from objectsin the scene. For example, the method may further include illuminatingthe scene with multiple narrow-band lights spaced at different frequencybands, and measuring intensity of reflected light for each frequencyband.

In another aspect, capturing the photogrammetric range data furthercomprises using at least one additional image-capturing componentcoupled to the camera to capture stereo photogrammetric data. In thealternative, or in addition, capturing the photogrammetric range datafurther comprises measuring time between emission of a signal from thecamera and detection of a signal reflection at the camera. Various otherways may also be used for capturing the photogrammetric range data. Forexample, capturing the photogrammetric data may include capturingadditional photographs using an aperture wider than the first apertureand different focal points. For further example, capturing thephotogrammetric range data may include illuminating the scene using alight source that spreads a known spectrum in a defined pattern.Capturing the photogrammetric range data may further include using adual shutter trigger to capture separate photographs, or using the lensin combination with a sensor array configured to detect photogrammetricmeasurement light at two or more defined points in a light path fromphotographed object to an image sensor of the camera.

In related aspects, a digital camera with optical elements andprocessing elements may be provided for performing any of the methodsand aspects of the methods summarized above. An camera may include, forexample, a camera body supporting a lens and a range finding sensorcoupled to a processor and a memory, wherein the memory holdsinstructions for execution by the processor to cause the cameraapparatus to perform operations as described above. An article ofmanufacture may be provided, including a non-transitorycomputer-readable medium holding encoded instructions, which whenexecuted by a processor, may cause a camera apparatus to perform themethods and aspects of the methods as summarized above.

Further embodiments, aspects and details of methods, apparatus andsystems for simulating a large aperture lens are presented in thedetailed description that follows.

BRIEF DESCRIPTION OF THE DRAWINGS

The present technology, in accordance with one or more variousembodiments, is described in detail with reference to the followingfigures. The drawings are provided for purposes of illustration only andmerely depict typical or example embodiments of the technology. Thesedrawings are provided to facilitate the reader's understanding of thetechnology and shall not be considered limiting of the breadth, scope,or applicability of the technology.

FIG. 1 is a block diagram showing an example of a system for simulatinga large aperture lens.

FIG. 2 is a flow chart illustrating a method for simulating a largeaperture lens.

FIGS. 3-9 are flow charts illustrating various additional aspects ofcapturing photogrammetric range data for simulating a large aperturelens.

FIG. 10 is a block diagram illustrating an example of an apparatus forsimulating a large aperture lens.

DETAILED DESCRIPTION

The present technology takes advantage of the higher light sensitivityof modern sensors, reducing lens aperture size without simultaneouslygiving up the ability to deliver the narrow depth of field that makeslarge aperture photographs desirable. Additional benefits may includeproviding the ability to elect the depth of field and the type andintensity of blurring for objects that are in the area that would be outof focus with a wider aperture. The present technology enables a cameraoperator to make such an election after the photograph has been taken.In addition, the present technology may enable simulated aperturechanges, for use with lenses designed with a fixed aperture. Fixedaperture lenses may provide reduced cost and reduced risk of breaking.

Various methods may be used to encode distance from the camera to thevarious objects in the field of view. Once the distance information isdetermined, it may be stored with the recorded image, such as within thephoto file or in an associated auxiliary file. In certainimplementations, the distance information may be embedded within thephotograph light and/or color data itself. The distance information mayadditionally or alternatively be used to alter the depth of field duringthe in-camera post-processing. Multiple variants may be produced(in-camera or during later processing) using varieties of combinationsof simulated apertures and focus points. To maximize the benefits ofthis technology while obtaining a desired shutter speed, algorithms maybe included that shut down the physical aperture to the smallest sizepossible while delivering the target shutter speed.

In an embodiment, as the light requirements of the sensor decrease, itis possible to utilize a lens with the focal characteristics of apinhole camera, allowing an infinite depth of field. Another embodimentmay use an autofocus point generally within the range of where thecamera would normally determine autofocus to best focus on. Anothervariant identifies a plurality of likely desirable focal points andselects an aperture that renders all of those focal points with clarity.Use of a pinhole type lens eliminates the need to account for extantlack of focus in adding the blur required to simulate a wider apertureor different focus point; without such a pinhole lens, the software mayneed to account for extant blur in adding new blur. In anotherimplementation, the camera “brackets” the focus on more than oneexposure in a manner that delivers an in-focus image throughout theentire scene, spread of multiple images. Such images are then utilizedin concert in creating the simulated focus point and aperture.

The type of aperture (number of blades, for example) dramaticallyimpacts the quality of the “bokeh”, or out-of-focus blur effect. Thesimulated aperture is advantageously selected to complement the existingbokeh in a non-pinhole, non-infinite focuses photograph, so thatmultiple bokeh types do not co-exist. In an embodiment, the user isallowed to select the simulated aperture type, and/or the camera orpost-processing software selects the simulated aperture type based upona rule set. In another implementation, the camera or post-processingsoftware selects a simulated aperture type and/or size and/or othercharacteristics to match a real world lens other than the lens actuallyused—for example, making an inexpensive F/8.0 fixed aperture lens renderphotographs that emulate those taken with a Canon F/2.8 70-200L II lens.In an embodiment, the camera and/or post-processing software contain adatabase of which lenses are capable of rendering images that canemulate which other lenses (e.g., “lens ‘A’ can emulate the following 24lenses” (list)).

To facilitate measuring range data, a specific range of light, forexample ultraviolet, infrared, or a very narrow band of the visiblespectrum (for example 415-416 nm) may be used. It may be advantageouseven with non-visible portions of the spectrum to use as narrow a bandas possible, to prevent noise or interference. It may also beadvantageous to have the camera or other sensor select one of aplurality of light frequencies based upon the relative absence ofambient light in the chosen frequency. It may also be advantageous toutilize a plurality of light frequencies so that the cleanest overallfrequency, or the cleanest frequency for each of a plurality of areas inthe image, can be used. The one or more light frequencies may begenerated by a camera mounted device, such as a flash, but may also begenerated off-camera, for example by an auxiliary flash unit. Becauselight intensity drops off with increasing distance from the lightsource, the intensity of the light reflected back at the camera is aprimary indicator of distance from the camera. However, because lightreflected off of certain surfaces is reflected more efficiently thanfrom other surfaces, it may be advantageous to use additional indicia ofdistance. Using a plurality of separate light frequencies, or evenutilizing radiation frequencies that are well outside of the visiblespectrum, can be used to determine drop-off due to reflectiondifferences. For example, an averaging mechanism may be utilized. Inaddition, the specific qualities of various portions of the radiationspectrum, such as the differences in the speed with which the radiationfall-off takes space at the long versus short end of the spectrum, canbe used to determine which measured variations are due to distance, andwhich to differences in reflection.

In an embodiment, contrast data for the visible light within the normalimaging range of the camera is combined with contrast data for the lightused to measure distance in order to determine the edges of objects.Objects within a framed area (such as a black and white striped shirt,which may be recognized as part of a single object despite the drop-offof light reflected from the black areas) can be treated as being at asimilar distance. In addition, the light data from the sensor can beused to adjust the distance data gleaned from the measurement of thereflected radiation (for example, areas that are darker in thephotograph may naturally reflect less light than lighter areas, and thedistance data for those areas may need to be increased to account forthe absorption the color difference would normally generate).

Where the radiation frequency used for the measuring guide light iswithin the spectrum visible to the camera, the post-processing mechanismadvantageously removes any light in that frequency. Alternatively, afilter placed over the lens, within the lens, or over the sensor may beused to prevent that frequency from being visible, although a secondarysensor, or a series of photographs, at least one without the sensor, maybe required in such a case. In any case where a portion of the visiblespectrum is removed, an embodiment the camera apparatus may repopulatethe missing portion of the spectrum by averaging the neighboring,unfiltered portions of the spectrum.

In an embodiment the camera apparatus may take photographs in sequence,where the light used to measure distance is released only for one of theplurality of photographs. Optionally, the “measuring photo” in thesequence may utilize a special filter over the sensor or within or overthe lens.

In a simple example, a photographer might mount a fixed f/11 lens andtake a photograph of a person in a field of flowers. The camera maytrigger a flash (in some implementations, using a laser to obtain theprecise frequency desired). The same or a different flash may projectlight used for other purposes, such as lighting up portions of the imagefor better imaging. The depth detection flash (whether integrated in aregular flash, embodied within one or more LEDs, generated with adiffused laser, or otherwise) may trigger during the exposure, releasingradiation at 380 nm and 720 nm. The selected light range must accountfor, and fall outside of, any filters between the scene and the sensors,and must also not fall outside of the sensor's range. The sensor maythen record the light falling on various portions of the sensor. In anembodiment, care is taken to avoid chromic aberration or otherprism-type optical imperfections that may cause light in the distancesensor range from landing on a different portion of the sensor thanvisible light reflected from the same object. To minimize the impact ofsuch chromic aberration, and to permit more accurate correction of sucherrors in post-processing (even without utilizing the other features ofthese embodiments), the measurement signal may include extremely narrowband signals on a plurality of wavelengths within the sensor range,advantageously at least one each toward the large and small ends of thevisible range, to allow use of the differential focus of the variousknown light components in the specific wavelength pulse to beinterpolated for the various wavelengths and used to correct the opticalerrors.

The relative strength of the 380 nm and 720 nm light (from the example)over the various portions of the photograph are used to determinedistance from the camera for the objects within the field. Objects thatdo not reflect one of the frequencies properly (i.e. an object thatabsorbs 380 nm radiation efficiently) are advantageously measured usingthe other frequency. Objects that absorb the plurality of frequencies(or objects that absorb the single frequency in the event of a systemimplemented using only one frequency) are examined digitally and, wherecontrast, light, and color indicate they are too distant to havereflected the light, they are marked as out of range and/or distant.Where the indications are that they are due to another factor, such asabsorption, their distance is imputed based upon the distance ofcomparable objects, nearby objects, and contrast.

The technology is advantageously implemented utilizing a single lightsensor, but may be implemented using a secondary sensor. In such a case,the secondary sensor is advantageously customized for the application,such as be being sensitive to the frequencies likely to be used and byhaving a low pass or other filter either absent, or configured to let inat least the frequencies likely to be used.

The use of light to measure distance is advantageously used incombination with the other mechanisms described herein, but need not be.

A second mechanism to obtain the necessary distance/depth information isto utilize at least one additional sensor to generate athree-dimensional image. The distance between the camera and the objectsmay be computed by using standardized mechanisms for measuring distancewhen binocular (or more than binocular) images are available.

An additional mechanism to obtain the necessary distance/depthinformation is to utilize the transmission of particles or waves inconjunction with measurement of their return reflection. In anembodiment, the sound of the shutter is used as the source of a sound,the reflection of which may be measured using a plurality of microphonesto generate a 3D map of the field of view of the camera. A secondarysound source may also be used.

With sound, and even with light, it is also possible to measure the timetaken from exposure to return and determine distance in this manner. Thespeed measurements may be used in addition to, or in place of, theintensity measurements described herein.

An additional mechanism to obtain the necessary distance/depthinformation is to take at least one, but advantageously a plurality, ofphotographs with a wider aperture and with different focal points. Forexample, the camera automatically, upon actuation, might set itsaperture to F/4 and focus to infinity and take a photograph, to 10 feetand take a photograph, to as close as possible and take a photograph,and then set its aperture to F/22 and focus to 20 feet (at thataperture, creating a nearly infinite focus) and then take the photographintended for use. In post-processing, other than objects moving in theframe between photographs (and the photographs can be taken in veryrapid succession, since even fast, phase-based autofocus isunnecessary), the relative blur of the objects in the infinite,mid-range, and macro focus photographs can be used to determine theirdistance from the camera. The F/22 photograph may then be post-processedto emulate any focal point and aperture because full distanceinformation is available.

In an embodiment, at least one secondary imaging device (such as a lensand sensor) is present on the camera and is used to generate at leastone image utilized to reconstruct focus. In an embodiment, a secondary,small lens with a fixed wide aperture relative to sensor size is used,advantageously in conjunction with a light-splitting method (such as asemi-translucent mirror), to take one, or advantageously a plurality of,photographs simultaneously or nearly simultaneously with the primaryimaging mechanism, where the placement of the secondary lens and/orsensor and/or configuration of the lens and/or sensor creates a knownfocal distance or depths. These may be implemented as a plurality ofsecond imaging units, rather than a single light-splitting unit orsequentially focusing/firing unit. For example, a DSLR might have threeadditional F/4.0 lenses and sensors mounted on the body, each aimed at a2 megapixel sensor, one set to infinity, one set to 2 feet, and one setto 15 feet for focus. Although the primary sensor may be 21 megapixels,the distance data sensors may generate a set of images that may becompared to the primary image and used to determine the correspondingobjects, and therefore the corresponding distances, of those objects.

The distance information can also be used to dynamically increase theamount of gain, brightness, or raw-processing “exposure” for areas basedon distance from a flash or other light source.

A brief description of some of the technology herein is the use of aflash, emitting in a known portion of the visible or non-visiblespectrum, and measure the time for the light to return and/or theintensity of the return. Record the time or intensity for each pixel orfor groups of pixels or areas, and then use that to generate distances.The post-processing software can then blur or sharpen or color orisolate based on distance.

An additional element is to utilize a light source that transmits onmore than a single wavelength (though none are necessarily visible) butwhich releases the light in a known spread pattern. For example, a lightsource with a prism might emit light from 350 nm to 399 nm, but theprism, and in some implementations a focusing device, may deliver thelight in a fixed pattern so that the light is spread in a predictableway across the field. Using a simple example of a light ranging from 381to 390 nm, the top 10% of the flash might be 390 nm, the bottom 10% ofthe flash 381 nm, and the remainder spread, increasing between 382 to389, over the other 80% of the field (although the top-to-bottom patternmay be left to right, for example, as well). Objects in the foregroundmay have a far larger range of light painting than objects in thebackground. In an embodiment, light painting may utilize a plurality oflight sources, each broadcasting on a different set of frequencies andplaced in different locations, in order to create what is effectively a3D light painting over the objects in the field of view. A simplemeasurement based on the areas shaded from one light source but not theother may provide significant depth information even without using morethan one light frequency per source. Use of varying frequencies fromeach of one or more sources may allow determination, in post-processing,as to where the object was relative to the camera and light source. Forexample, a camera pointed straight across a room, where a light sourceat the top of the room illuminates down, at a 45 degree angle, across aspectrum (say 399 nm at the ceiling, 390 on the floor), may captureinformation permitting a determination of whether an object was small,and close to the camera, or large and distant from the camera, dependingon how much of a light frequency spread is present on the object.

In certain digital light sensors, a light-splitting mechanism, such as aprism, is used to direct certain light frequencies toward certain areasof the sensor, typically functioning in such a manner that one pixelreceives red, another green, and another blue. The true color of eachpixel is reconstructed from this data, for example by interpolating thecolor (for example, the amount of red and blue present in the pixel thathas received only green light on the filter) based on neighboringpixels. One implementation may include filtering (or splitting via alight-splitting mechanism) all light hitting certain pixels except forthe one or more frequencies used to recreate distance data. The samemechanisms utilized to reconstruct missing color data in the RGBlight-split sensor design can be utilized to reconstruct the true colordata for the measurement pixel locations.

Another mechanism is to utilize a dual shutter trigger, wherein theshutter triggers once for the actual photograph and then immediatelybefore or after for a photograph measuring distance. Optionally, a lowpass or other filter may be deployed for one or both exposures. In anembodiment, in cameras with a physical shutter (such as a mirrorDSLR-style shutter), the initial photograph is taken first, and theshutter is digitally closed (or at least the data retrieved from thesensor), then the measurement light is triggered and a second exposurewith the measurement light is utilized. Because the measurement lightexposure utilizes a fixed, known source and quality of light (i.e. aflash sending light at a set frequency or frequencies), the secondexposure need not match the speed (or aperture) of the first exposure.Advantages in correlating the two images may exist if the sensor isread, but not discharged, between exposures so that the first data setis the sensor's normal exposure, and the second is the normal exposureplus the measurement light exposure. Advantageously the second exposureis very short. In an embodiment, the strongest measurement light sourceavailable that does not cause potential harm to vision or other harm isutilized in order to keep the additional exposure time in the hundredthsor thousandths of a second. In an embodiment, when a flash is used toilluminate the visible light for the primary exposure, the measurementlight is sent and measured first if the flash is set for “firstcurtain”, and the measurement light is sent and measured second if theflash is set for “second curtain”. A “first curtain” flash illuminatesthe scene at the beginning of an exposure period that lasts longer thanthe flash itself. A “second curtain” flash illuminates the scene at theend of an exposure that lasts longer than the flash itself. By measuringdistance as closely in time to the firing of the flash, the distancedata for moving objects will match those most visible in the photograph.Where the measurement signal is read before the primary exposure, it maybe advantageous to discharge the data from the sensor prior to beginningthe primary photograph.

In another embodiment, movement within a single frame may be measured bytriggering a measurement signal before or at the beginning of anexposure and by triggering a second measurement signal (either with thesensor discharged between exposures, or by utilizing differentwavelengths for each of the plurality of measurement signals). More thantwo measurement signals may be used, particularly for longer exposures.By comparing the distance data present at the start and end of thephotograph, the movement of objects may be measured. In a variant, aplurality of photographs is taken (with or without measurement signals)and using standard mechanisms are used for identifying elements withinphotographs (in combination with distance measurement data in anembodiment). The measurement of the movement of objects is then utilizedto reverse some of the motion blur, to improve or enhance the motionblur, or to select the blurred areas for other processing, either in anautomated fashion or by exposing the selection to a user who may thenfurther manipulate the area (or the non-motion areas) by utilizingPhotoshop™ filters or similar techniques.

Using the multiple exposure technique just described to identify motionblurred areas, it is possible to correct motion blur caused by ahandheld or other error causing the camera to move relative to the lightsource in a manner that causes a blur (such as by attempting tohand-hold a shot with too long an exposure time). The additional shotsused for motion detection need not be properly exposed, may utilizenon-visible portions of the light spectrum, and may utilize anartificial light source not present in the primary photograph. Forexample, a flash may trigger at 380 nm with substantial power; a veryrapid shutter photograph may be taken, followed by a 1/10 of a secondvisible light exposure, followed by a very rapid shutter photographusing another trigger of the flash at 380 nm. Because the flash may notbe visible to humans, even a very bright flash may not be disruptive,and may allow the position of objects immediately before and after theprimary exposure to be measured. In this manner, moving objects and evenoverall blur caused by poor technique or too slow a shutter speed for ahand-held shot may be digitally reversed, identified, enhanced, orotherwise manipulated. In a variant, a long exposure with a briefvisible light flash may be used in conjunction with a single flash(visible or otherwise) instead of triggering two measurement flashes(i.e. a first curtain flash replaces the first measurement flash, or asecond curtain flash replaces the second measurement flash). Themeasurement flash may advantageously be in a non-visible spectrum area,but in the alternative, or in addition may also be visible. Anunderexposed photograph with ambient light, a weak flash from theprimary flash used for the main exposure, or from another light source,may be used in place of one or more of the measurement exposures.

The amount of movement between exposures, advantageously in combinationwith recording the data regarding the length of the exposures and thetime between exposures, may be used to determine speed of the object(s)being photographed.

Using a simple example of a tennis ball flying past a spectator, thepre-exposure and post-exposure measurement photographs may identify thatthe tennis ball is moving at 100 feet per second. If the primaryexposure was 1/30th of a second but the photographer wants to enhancemotion blur, the photographer may tell the system to predict theposition the ball may have occupied at 1/15th of a second exposure, andthe software may then lengthen the blur to reflect that new position.Alternatively, the photographer may wish to reduce the motion blur. Thecalculated position of the tennis ball at a 1/250th of a second exposuremay be used to shorten the motion. Technology such as Adobe Photoshop's™“context-aware fill” may be used to fill the area obscured by thenow-eliminated blurred portion.

It should be noted that the use of distance data in conjunction withsoftware that requires or benefits from selection of specifiedphotographic elements allows for far more accurate selection. Forexample, it is widely acknowledged that selecting curly hair in aphotograph is very difficult, as the hair overlies many differentbackground elements. Using the distance data to select only elementswithin a few inches of the head should result in a selection of thehair, head and body only. The selection can then be modified intraditional ways. Similarly, a white bird sitting on a light post infront of white clouds may be nearly impossible to select or isolate inan automated manner, but use of distance information allows automatedselection of the bird. Distance information may be used alone, inconjunction with traditional tools, in conjunction with manualselection, or in any combination thereof.

An additional implementation is to identify the drop off in flashillumination and use that drop off to identify distance information.Particularly where the white point of the flash differs from the whitepoint of the ambient light, the light quality differential,advantageously measured against similar objects, can be used todetermine distance.

Another method for generating distance data, whether for encoding suchdata for use in generating 3D images, for directing generating 3Dimages, or for manipulating focus point, assisting, enhancing, oraccomplishing the selection of specified image elements, or otherwise,is to utilize a single lens in combination with a sensor capable ofdetecting light at two or more points in the light's path from object tofilter. This may take the form of a stacked set of filters with thefirst filter or filters being partially transparent. This may also takethe form of a semi-transparent mirror or a light splitting mechanism, ora combination, which functions to direct the image to at least twosensors that measure the photons (or other radiation) after the photonshave traveled a different distance to each of the filters.Alternatively, or in combination with other implementations, one or morefilters may be at least partially reflective, with the reflected lightmeasured by a second filter. Advantageously, the reflective filter iseither angled toward the second filter or toward an additionalreflective surface that points to a second filter. Alternatively,reflective elements may be integrated into the filter where suchelements point to a second filter or second reflective surface. In anembodiment, the distance between the sensors may be modified to provideoptimal quality and quantity of distance data. For example, with anextremely narrow depth of field, it may be necessary to locate thesensors very close to each other in order to avoid having the lighthitting the sensors being focused so differently that it becomesdifficult to correlate. Similarly, with a wide depth of field, or withvery distant objects, placing the sensors further apart may be desirableto gain additional depth data. In an embodiment, the distance betweenthe sensors may be adjusted automatically, manually, or continuouslybased on data from the camera, the operator, or other sources.

Referring to FIG. 1, a camera assembly according to a dual sensorembodiment is shown. A camera body 102 including a housing holds a lensassembly 104 in an optical path (light cone) 115 for photographing asubject 110. It should be appreciated that the lens 104 typicallyincludes several optical elements and bends the light cone 115 includingrays 112, 114 in complex ways, but such features and effects are omittedfrom FIG. 1 for illustrative simplicity. Similarly, typical complexitiesof camera bodies are also omitted from the diagram.

A transmissive (e.g., semi-transparent) first sensor 106 andnon-transmissive second sensor 108 may be placed in line with the path(light cone) 115 that light follows from the subject 110 through thelens 104, so that a sufficient amount of light is measured at the firstsensor 106, and a sufficient amount of light to measure at the secondsensor 108 passes through the first sensor 106 to the second sensor 108,where it is measured. Groups of photons reflected from areas on thesubject 110 that are sufficiently similarly situated so as to beindistinguishably distant from each other given the resolvinglimitations of the optics in the lens 104 or the sensors 106, 108 followwhat are, for practical purposes, identical paths. Two illustrativephoton paths or rays 112, 114 are shown. It should be understood thatmultiple photons may be reflected or emitted simultaneously fromindistinguishably close areas on the subject 110 and that the shutter orcamera sensor discharge function will almost certainly be too slow toprevent multiple photons from being reflected at different times fromthe same location. Light from a photon path 112 passes through the lens104 where it is focused onto the sensors 106, 108. The lens 104 may beconfigured to achieve desired focus on one of the sensors 106, 108,which may be considered the primary sensor. To achieve sharp focus onboth sensors 106, 108, which may require a narrowing of the aperture inthe lens 104, one or more additional sensors (not shown) may be added sothat the focal point is set to a sensor located between at least twoother sensors 106, 108.

When the camera is configured properly for the photograph, data isreceived from the sensors 106, 108 and provided to one or moreprocessors 116. The one or more processors 116 may generate image dataand metadata as described herein, and store the image data and metadatain a memory 118. Using the ray 112 as an example, the photons havetraveled along a path 112 through the lens 104 which focuses them ontoone of the sensors 106, 108 or other focal plane. Some of the photonsare measured at the first sensor 106 and are recorded at a firstinterception point 120, while others pass through the first sensor 106to the second sensor 108 and are measured at a second interception point122. While in an embodiment the camera apparatus may avoid additionaloptical manipulation of the photons between the sensors, suchmanipulation may be necessary or desirable for various reasons, forexample to filter out certain spectrum portions or to artificiallycreate a similar photon path as may exist with greater or lesser thanactual distance between the sensors.

By comparing the qualities of the light measured at each of theplurality of sensors and correlating the light measured at each of thesensors to determine which pixels on each sensor are receiving lightreflected from the same object in the field of view, one element of thetechnology is to enhance the focus or other qualities of the photograph.Using the diagram, the light reflected from a single point on an objectalong photon path 112 falls on one intercept point 120 on the firstsensor 106 and another intercept point 122 on the second sensor 108.Software may be used to measure the contrast between adjacent pixels,the intensity, white point, color, and/or wavelength of the light on thepixel, any or all of which may be used in conjunction with analysis oflight measured on other areas of the sensors. In this way, it can bedetermined that the light falling on each of the intercept points 120,122 originated at the same source. By comparing the locations of theintercept points 120, 122, advantageously also with the distance data(generated as set forth herein), it is possible to determine the path112 that the light traveled to reach the sensors by ray tracing.

Further, by comparing the light qualities at the intercept points 120,122 and surrounding pixels, advantageously in light of aperture andfocal point, and focal length data, the technology may include comparingthe relative sharpness of the focus in light of the predicted sharpnessof focus for objects at various distances in order to determine distanceof the object reflecting the light. For example, if the first sensor 106intercept point 120 shows a black line four pixels wide with sharpmargins surrounded by white, but the second sensor 108 intercept point122 shows a black line 4 pixels wide, a gray margin 2 pixels wide, andthen a white surrounding area, the portion of the subject 110represented by the light traveling along the photon path 112 is in focusat the first sensor's intercept point 120 but out of focus by ameasurable amount at the second sensor's intercept point 122. Becausethe sensors are located at different distances along the photon path112, the one or more processors 116 can work backwards from the focusdata to determine distance to the subject 110.

Another implementation may include utilizing the different focal datafor the various intercept points reflecting light falling along each ofthe photon paths in order to permit more accurate noise reduction (forexample, by removal of noise that appears only on one of the pluralityof sensors) or to permit more accurate post-image-capture focusenhancement. In one implementation, the amount of blurring that iscaused by an increase in the focal length by the distance between twosensors 106, 108 can be utilized to differentiate between an actualobject in the photograph (which should not be enhanced or sharpened) anda focal blur artifact, which should be. The amount of sharpening shouldadvantageously be altered for various areas in the photograph to matchthe amount of focal blur artifact detected in each area.

The present technology includes utilizing the depth data, howevergathered, to enhance or facilitate digital reconstruction of the imageas it may have looked if captured by two separate cameras (for example,a 3D camera with two lenses set apart by a similar distance to humaneyes). Some alternative embodiments may include providing two or moreimaging sensors and utilizing the data from the additional sensor orsensors to generate the image from the second position. The secondsensor(s) need not have the same resolution, color range, lightsensitivity or other qualities; in such a case, data from at least theprimary sensor can be utilized to enhance the data from the secondarysensor.

In another variant, a plurality of sensors may be intentionally utilizedwith differing characteristics (either in terms of sensor limitations orthe settings used for the sensor for that image). For example, a 3Dcamera with right and left lenses and corresponding right and leftsensors may have the right sensor set to high light sensitivity (andthus high noise artifacts) and the left sensor set to low lightsensitivity (and thus low noise artifacts). The data from each sensormay be used to enhance the data from the other, for example byidentifying and removing artifacts present in one but not the other,adding back color data to the low light sensitivity image but present inthe higher light sensitivity image, or similar functions. This isparticularly useful in generating HDR, or high dynamic rangephotographs. A variant useful in some situations, and which provides alarger advantage over simply processing the same raw data from a singlesensor multiple times, is to vary the aperture and/or shutter speedbetween the two photographs. While the resulting image may need to becorrected to create or remove focal or motion blur to match the twoimages, it may benefit from delivering different amounts of light toeach sensor. The sensors and lenses may also be different sizes, so thata smaller lens pointing to a smaller first sensor, optionally with thesame number of pixels as the larger lens pointing to the larger secondsensor, will naturally deliver more similar focal and speedcharacteristics than attempting to use optics and speed to deliver thesame image to identically sized sensors.

The foregoing examples and details may be embodied in one or moremethodologies performed by a digital camera including an image sensor,processor and memory. Methodologies that may be implemented inaccordance with the disclosed subject matter will be better appreciatedwith reference to various flow charts, summarizing more detailed aspectsof the methodologies described above. Although methodologies are shownand described as a series of acts/blocks for simplicity of illustration,it is to be understood and appreciated that the claimed subject matteris not limited by the number or order of blocks, as some blocks mayoccur in different orders and/or at substantially the same time withother blocks from what is depicted and described herein. Moreover, notall illustrated blocks may be required to implement methodologiesdescribed herein. It is to be appreciated that functionality associatedwith blocks may be implemented by software, hardware, a combinationthereof or any other suitable means (e.g., device, system, process, orcomponent). Additionally, it should be further appreciated thatmethodologies disclosed throughout this specification are capable ofbeing stored as encoded instructions and/or data on an article ofmanufacture, for example, a non-transitory computer-readable medium, tofacilitate storing, transporting and transferring such methodologies tovarious devices. Those skilled in the art will understand and appreciatethat a method may alternatively be represented as a series ofinterrelated states or events, such as in a state diagram.

As shown in FIG. 2, a camera may be used to perform a method 200 forsimulating a large aperture lens. The method 200 may include capturing aphotographic image data of a scene using a camera having a lens set to afirst aperture size. At 204, concurrently with capturing the image, thecamera may capture photogrammetric range data pertaining to depth ofobjects in a field of view of the camera. As used herein, “concurrently”means simultaneously, or within a time window that is small enough tominimize uncontrolled changes in the photographed scene (for example,less than one 1 millisecond for still photographs, or less for actionphotographs). Various techniques for capturing photogrammetric rangedata are described in the foregoing detailed description, and summarizedbelow in connection with FIGS. 3-9. A processor in the camera, or aremote processor, may perform the element 206 of processing thephotographic image data using the photogrammetric range data to obtainsecond photographic data simulating an image captured using a lens setto a second aperture size, wherein the second aperture size is widerthan the first aperture size. Various processing algorithms forprocessing the image data using the range data to obtain the secondphotographic data are described in the detailed description above. Theprocessor may further perform the operation 208 of storing thephotographic image data (e.g., a digital photo) associated with themeasurement data in an electronic memory.

With reference to FIGS. 3-9, several additional operations 300-900 aresummarized for capturing photogrammetric range data, which may beperformed by the camera apparatus, alone or in cooperation with one ormore auxiliary sensor or device. One or more of operations 300-900 mayoptionally be performed as part of method 200. The elements 300-900 maybe performed in any operative order, or may be encompassed by adevelopment algorithm without requiring a particular chronological orderof performance. Operations can be independently performed and are notmutually exclusive. Therefore any one of such operations may beperformed regardless of whether another downstream or upstream operationis performed. For example, if the method 200 includes at least one ofthe operations 300-900, then the method 200 may terminate after the atleast one operation, without necessarily having to include anysubsequent downstream operation(s) that may be illustrated.

In an aspect, the method 200 may include the additional operations 300for capturing photogrammetric range data, as shown in FIG. 3. The method200 may include, at 302, illuminating the scene using a narrow-bandlight adjacent to the lens, and measuring intensity of narrowband lightreflected from objects in the scene. The method may further include, at304, illuminating the scene with multiple narrow-band lights spaced atdifferent frequency bands, and measuring intensity of reflected lightfor each frequency band.

In an aspect, the method 200 may include the additional operation 400for capturing photogrammetric range data, as shown in FIG. 4. The method200 may include, at 400, using at least one additional image-capturingcomponent coupled to the camera to capture stereo photogrammetric data.For example, a specialized flash unit and/or narrow-band emitter may becoupled to the camera to generate a range-finding signal.

In connection with using a range-finding signal, the method 200 mayinclude the additional operation 500 for capturing photogrammetric rangedata, as shown in FIG. 5. The method 200 may include, at 500, measuringtime between emission of a signal from the camera and detection of asignal reflection at the camera. The signal reflection may be identifiedby being of a particular narrow-band frequency that is not generallypresent in the lighting environment, including, for example anultraviolet, infrared, high-frequency sound, or other reflection.

In an aspect, the method 200 may include the additional operation 600for capturing photogrammetric range data, as shown in FIG. 6. The method200 may include, capturing additional photographs using an aperturewider than the first aperture and different focal points. Thephotographs may be captured in very rapid succession so as to minimizechanges due to movement of the subject or changes in the lightingenvironment. If the operation 500 is used, this may require awider-aperture lens and negate the advantages of using a small-aperturelens. However, the benefits of adjusting a focal point inpost-processing anytime after an image is copied are retained.

In another alternative, the method 200 may include the additionaloperation 700 for capturing photogrammetric range data, as shown in FIG.7. The method 200 may include, at 700, illuminating the scene using alight source that spreads a known spectrum in a defined pattern. Forexample, a light source with a prism might emit light from 350 nm to 399nm, but the prism, and in some implementations a focusing device, maydeliver the light in a fixed pattern so that the light is spread in apredictable way across the field. Further details are provided in thedetailed description above.

In another alternative, the method 200 may include the additionaloperation 800 for capturing photogrammetric range data, as shown in FIG.8. The method 200 may include using a dual shutter trigger to captureseparate photographs. In this technique, the shutter triggers once forthe actual photograph and immediately before or after for a photographreserved for measuring distance. Optionally, a low pass or other filtermay be deployed for one or both exposures. Further details are providedin the description above.

In an aspect, the method 200 may include the additional operation 900for capturing photogrammetric range data, as shown in FIG. 9. The methodmay include, at 900, using the lens in combination with a sensor arrayconfigured to detect photogrammetric measurement light at two or moredefined points in a light path from photographed object to an imagesensor of the camera. This may include the use of dual image sensors,for example as described above in connection with FIG. 1. Alternativesensor arrangements for detecting photogrammetric measurement light usedfor range finding may also be used.

With reference to FIG. 10, there is provided an exemplary apparatus 1000that may be configured as digital camera, for capturing image data in amanner enabling post-processing to simulate a large aperture lens. Theapparatus 1000 may include functional blocks that can representfunctions implemented by a processor, software, or combination thereof(e.g., firmware).

As illustrated, in one embodiment, the apparatus 1000 may include anelectrical component or means 1014 for capturing a photographic imagedata of a scene using a camera having a lens set to a first aperturesize, coupled with or embodied in a memory 1004. For example, theelectrical component or means 1002 may include at least one controlprocessor 1002 coupled to a memory component 1004. The control processormay operate an algorithm, which may be held as program instructions inthe memory component. The algorithm may include, for example, focusing alens mechanism, recording an aperture setting, triggering operation of ashutter mechanism, and recording data from an image sensor.

The apparatus 1000 may further include an electrical component or module1016 for, concurrently with capturing the image, capturingphotogrammetric range data pertaining to depth of objects in a field ofview of the camera. For example, the electrical component or means 1016may include at least one control processor 1002 coupled to the memorycomponent 1004. The control processor may operate an algorithm, whichmay be held as program instructions in the memory component. Thealgorithm may include, for example, any of the more detailed algorithmssummarized in connection with FIGS. 3-9, or described elsewhere herein.

The apparatus 1000 may further include an electrical component or module1016 for processing the photographic image data using thephotogrammetric range data to obtain second photographic data simulatingan image captured using a lens set to a second aperture size, whereinthe second aperture size is wider than the first aperture size. Forexample, the electrical component or means 1016 may include at least onecontrol processor 1002 coupled to the memory component 1004. The controlprocessor may operate an algorithm, which may be held as programinstructions in the memory component. The algorithm may include, forexample, tracing light rays measured by an apparatus as shown in FIG. 1to determine how the image should appear for one or more focus points,if the image had been collected via a wider aperture lens, therebyadding bokeh not present in the original image. In an alternative, bokehmay be added to an image gathered using a small aperture lens, forexample, by defining a focal plane based on a based on virtual camerasetting (e.g., including aperture size), determining which pixels are inthe focal plane, and blurring pixels that are out of the focal plane inan amount proportional to the distance from the focal plane. This maysimulate bokeh without requiring sophisticated ray tracing, which may betoo computationally intensive.

The apparatus 1000 may include similar electrical components forperforming any or all of the additional operations 300-900 described inconnection with FIGS. 3-9, which for illustrative simplicity are notshown in FIG. 10.

In related aspects, the apparatus 1000 may optionally include aprocessor component 1002 having at least one processor, in the case ofthe apparatus 1000 configured as a video processing apparatus, alone orin combination with a client device. The processor 1002, in such casemay be in operative communication with the components 1014-1018 orsimilar components via a bus 1012 or similar communication coupling. Theprocessor 1002 may effect initiation and scheduling of the processes orfunctions performed by electrical components 1014-1018.

In further related aspects, the apparatus 1000 may include sensorcomponent 1006 for measuring photogrammetric range information or animage collected using a small aperture lens. The apparatus may includean input-output device 1008, for example, a touchscreen or keypad forreceiving user input and displaying a user interface and capturedphotographs. The apparatus 1000 may optionally include a separatecomponent for storing images, such as, for example, a memory card,non-transitory computer-readable medium or other non-volatile datastorage component 1010. The computer readable medium or the memorycomponent 1010 may be operatively coupled to the other components of theapparatus 1000 via the bus 1012 or the like. The memory components 1004and/or 1010 may be adapted to store computer readable instructions anddata for implementing the processes and behavior of the components1014-1018, and subcomponents thereof, or the processor 1002, or themethods disclosed herein. The memory components 1004 and/or 1010 mayretain instructions for executing functions associated with thecomponents 1014-1018. While shown as being internal to the memory 1004,it is to be understood that the components 1014-1018 can exist externalto the memory 1004. The apparatus may include other components commonlyincluded in digital cameras or other electronic devices (e.g.,smartphones, notepad computers, etc.) that include a camera function.

The steps of a method or algorithm described in connection with thedisclosure herein may be embodied directly in hardware, in a softwaremodule executed by a processor, or in a combination of the two. Asoftware module may reside in RAM memory, flash memory, ROM memory,EPROM memory, EEPROM memory, registers, hard disk, a removable disk, aCD-ROM, or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor such that theprocessor can read information from, and write information to, thestorage medium. In the alternative, the storage medium may be integralto the processor.

In one or more exemplary designs, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored on ortransmitted over as one or more instructions or code on a non-transitorycomputer-readable medium. Computer-readable media may includes bothcomputer storage media and non-transitory communication media includingany medium that facilitates transfer of a computer program from oneplace to another. A storage media may be any available media that can beaccessed by a general purpose or special purpose computer. By way ofexample, and not limitation, such non-transitory computer-readable mediacan comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage,magnetic disk storage or other magnetic storage devices, or any othermedium that can be used to carry or store desired program code means inthe form of instructions or data structures and that can be accessed bya general-purpose or special-purpose computer, or a general-purpose orspecial-purpose processor.

The previous description of the disclosed embodiments is provided toenable any person skilled in the art to make or use the embodimentsdisclosed. Various modifications to these embodiments will be readilyapparent to those skilled in the art, and the generic principles definedherein may be applied to other embodiments without departing from thespirit or scope of the disclosure. Thus, the present disclosure is notintended to be limited to the embodiments shown herein but is to beaccorded the widest scope consistent with the principles and novelfeatures disclosed herein.

What is claimed is:
 1. A photographic flash apparatus, comprising: oneor more light generation elements capable of generating light within anarrow frequency range, wherein the one or more light generationelements, when triggered, generate in sequence: a first light within anarrow frequency range; a second light within a visible frequency range,whether broad or narrow; and a third light within a narrow frequencyrange.
 2. The apparatus of claim 1, wherein the first and third lightsare generated with substantial power.
 3. The apparatus of claim 1,wherein both the first and third lights are outside of a portion of thelight spectrum visible to humans.
 4. The apparatus of claim 1, whereinone, but not both, of the first light or third light is outside of theportion of the light spectrum visible to humans.
 5. The apparatus ofclaim 1, wherein the apparatus is operably coupled to a camera and eachof the first, second, and third lights are capable of detection by theimaging apparatus within the camera.
 6. The apparatus of claim 5,wherein the camera is programmed to identify motion blur in objects inan image utilizing positions of the objects as illuminated by the firstand third lights, and corrects the image generated for motion blurutilizing the identified motion blur.
 7. The apparatus of claim 5,wherein the camera is programmed to identify motion blur in objects inan image utilizing the positions of those objects as illuminated by atleast two of the first, second, and third lights, and corrects the imagegenerated for motion blur utilizing the identified motion blur.
 8. Theapparatus of claim 5, wherein the camera is programmed to receive fromthe apparatus, the frequency or frequencies of at least two of thefirst, second, and third lights and store the frequency or frequencies.9. A photographic flash apparatus, comprising: one or more lightgeneration elements, wherein the light generation element(s), whentriggered, generate a standard flash, and one or both of a first lightcorresponding to a first curtain flash and a second light correspondingto a second curtain flash.
 10. The apparatus of claim 9, wherein one,but not both, of the first light or second light is outside of theportion of the light spectrum visible to humans.
 11. The apparatus ofclaim 9, wherein one, but not both, of the first light or second lightis within a narrow frequency range.
 12. The apparatus of claim 9,wherein both the first and second lights are triggered and are within anarrow frequency range.
 13. The apparatus of claim 9, wherein theapparatus is operably coupled to a camera and one or both of the firstand second lights are capable of detection by the imaging apparatuswithin the camera.
 14. The apparatus of claim 13, wherein the camera isprogrammed to identify motion blur in objects in an image utilizingpositions of the objects as illuminated by one or both of the first andsecond lights and positions of the objects as illuminated by a standardflash, and corrects the image generated for motion blur utilizing theidentified motion blur.
 15. The apparatus of claim 13, wherein thecamera is programmed to receive from the apparatus, the frequency orfrequencies of at least one of the first and second lights and store thefrequency or frequencies.
 16. An image editing apparatus, comprising acomputer and memory programmed to: read image data generated by a cameraoperably coupled to a flash, wherein the flash, when triggered, recordsimage data in a single image generated by at least two flashes, whereinat least one of the at least two flashes is within a narrow band; removethe portions of the image that fall within the narrow band; and replacethe portions of the image falling within the narrow band with datainterpolated from adjacent areas of the image.